Streaming data is a continuous flow of data that must be ultimately presented to a user in a particular sequence in real time. Digital samples representing an audio signal, for example, must be converted to a sound wave in the same sequence they were transmitted, and at exactly the time spacing they were generated, or some user-specified alternative. Digital data representing video frames require assembly into the proper sequence in the frame for presentation on a display together, and successive frames must display at the correct real-time rate.
Streaming data need not necessarily maintain correct sequence or timing throughout an entire communication chain among various transmitters, processors, memories, and receivers. Indeed, video and audio clips are frequently stored as static data in recording media, computer memories, and network buffers. Packet-switched systems might also carry parts of the same streaming data over different paths and even in different time sequences. Processors such as digital filters can assemble parts of the data stream, modify them as a static unit, then release them to further units in the system. Eventually, however, the stream must be heard or seen in the correct sequence at the proper relative times.
Streaming data almost always involves very large amounts of data. Streaming data almost always challenges the capacity of digital buses in computers to access it, carry it and switch it. Streaming data almost always taxes the processing power of functional units, both software and hardware, to receive it, convert it, and pass it on to other units. Those in the art speak of the necessity of “fat pipes” for streaming data.
An architecture called WDM-CSA (Windows Driver Model Connection and Streaming Architecture) introduces the concept of a graph for specifying the connections among the facilities of a computer where a data stream must pass through a number of processing units in an efficient manner. The WDM-CSA protocol also simplifies the development of drivers for such data. Basically, WDM-CSA specifies the flow of data frames through a graph, and also the control protocols by which adjacent modules in the graph communicate with each other to request and accept the data frames.
Commonly assigned patent application “Improving the Flow of Streaming Data through Multiple Processing Units,” filed on even date herewith (Ser. No. 09/310,610), introduces the concept of data pipes for enhancing the data flow of streaming-data frames through a graph of interconnected modules in WDM-CSA and in other streaming-data environments. Basically, data pipes avoid redundant storage and copying of data as a number of modules process the frames, and streamline allocation of the frames in which the data is packaged. Another commonly assigned application, “Improving the Control of Streaming Data through Multiple processors,” filed on even date herewith Ser. No. 09/310,587, presents a mechanism for controlling the flow of frames through multiple modules by improving the control throughout the entire graph, rather than by optimizing each individual module separately. These applications are incorporated by reference.
In many applications of streaming data, processing could be simplified and improved by adding capabilities for splitting one large frame into multiple subframes and for mixing multiple frames together into a single large frame. For example, multimedia presentations commonly have a single collection of data representing different modalities that are to be output together to a user. A frame of television-type data may represent an entire field of an NTSC signal, including video data on a number of scan lines, audio data, a picture-in-picture frame on parts of some of the scan lines, and data in a video blanking interval (VBI). VBI data might include character codes for close-captioning, digital data representing Web-TV information, and a variable number of other subfields. These frames require complex mixing of data when they are generated, and then must be split apart for processing by different hardware or software modules concurrently for presentation to a viewer. While the audio data in one subframe is filtered in tone controls and compressed/expanded in volume, the video data is resized and color balanced, the close-captioning is formatted and positioned, and so forth.
The systems described in the patent applications noted above describe primitive abilities for splitting large frames into smaller frames for separate processing. However, the growing need for multimedia and similar data streams having multiple types of data demands the further ability to combine, merge, or mix multiple frames into a single frame, and also requires efficient data flow and control of split and mixed frames of streaming data.